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LETTER FROM THE EDITOR 


Welcome to the Fall 2020 volume of the A/R 
Professional File! In this volume, you will find two 
articles whose authors use large databases from 
community colleges and statewide systems to 
address issues with significant policy implications. 


It seems that debates have raged for decades over 
how to use completion rates to evaluate college 
performance. No matter what formula is adopted, 
the resulting computations often fail to reflect our 
diverse missions. What if we used a student-focused 
methodology instead? In his article, Rethinking 
Completion Analytics to Better Support the Student 
Experience across Diverse Ecosystems, Jeffrey L. Cornett 
proposes an approach to completion rates that 
adjusts to the institution's ecosystem and range of 
students we serve. Check out his CAT-scan graphics! 


In their study Brain Drain in Maryland: Exploring 
Student Movement from High School to Postsecondary 
Education and the Workforce, Amber Bloomfield, 
Bess A. Rose, Alison M. Preston, and Angela K. 
Henneberger examine the migration of students 
who leave the state to attend college elsewhere. 
Which students are most at risk of not returning 

to the state workforce? Their analysis comprises a 
unique application of propensity score matching 


approach to better understand brain drain and 
mitigate its impact. 


Wherever and however you are working these days, 
| hope you are healthy and happy. Soothe the stress 
of this ongoing health crisis by treating your brain to 
some new ideas. Share yours with your colleagues 
by sending a manuscript to the A/R Professional File. 


Sharron Ronco 
Coordinating Editor 
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Abstract 


Single measures of college performance fail to 
reflect the mix and diversity of students served. 
First-time students vary widely in terms of their 
readiness to succeed in college. Higher education 
environments (ecosystems) also vary widely and 
afford students differing admission and transfer 
opportunities. These factors confound college 
completion analytics when those who define college 
goals focus only on what is best for the college. A 
fresh way to rethink completion analytics would 

be to measure college completion success from 

the student point of view and how they wish to 
experience college. Student-focused completion 
rate scorekeeping would improve research into best 
practices. Student advising would also improve if 
completion analytics were disaggregated along a 


continuum of readiness to succeed. 


Keywords: student experience, college readiness, 
CAT-scan graphs, college ecosystem, sister university, 
transfer rates, 1+3 transfers, 2+2 transfers, second- 
chance transfers, completion rates, vertical lift, right 


shift, institutional research, student advising 


1. The Central Indiana service area of lvy Tech Community College includes 
Indianapolis (Marion County) and eight surrounding counties (Boone, 
Hamilton, Hancock, Hendricks, Johnson, Morgan, Putnam, and Shelby). In 

2017 lvy Tech was reorganized statewide to formally eliminate the regional 
layer of management. The name “Ivy Tech” in this article refers to the Central 
Indiana region or service area of that college, unless otherwise noted. 


https://doi.org/10.34315/apf1502020 
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INTRODUCTION 


In my first meeting with our college’s new chief 
information officer | began to explain problems 
with completion performance scorekeeping. 
After patiently listening for a while, he interrupted 
to challenge me to prescribe the exact college 
scorekeeping formula that | proposed. At first, | 
did not know how to respond. Just changing the 
completion formula does not solve the problem. 
After pausing to think for a moment, | suggested 


that his question was the wrong one to ask. When 


judging success, it is not the college performance 


that matters—it is the student performance! We 
should try to analyze and improve how the student 
experiences college, and not merely how the college 


experiences students. 


PROBLEM STATEMENT 


Completion analytics that focus on the college 
experience rather than the student experience 


present us with three problems: 


1| Students’ goals might not match the college's 
goals. Colleges and their scorekeepers naturally 
want to experience students who have high 
rates of retention and completion. Students, 
however, when given the opportunity to 
transfer, might have different goals that make 


them choose to transfer before graduating from 


their first school attended. 


2| Student mix varies across colleges. When 
analyzing college performance, it is much 
too convenient to incorrectly assume that 


all colleges have the same mix of students 


operating within comparable higher education 


environments. This false assumption allows 


Fall 2020 Volume 


all differences in performance outcomes to 

be attributed to the college, and none to the 
students. This oversimplifies comparisons 
across colleges and the interpretation of results. 
It is too easy to presume that higher-scoring 
colleges must simply have better leadership, 
academics, or Support programs, and that 
lower-performing colleges must be doing 
something dramatically wrong or must be 


missing major improvement opportunities. 


3| Single measures of college performance can 
mislead students and their advisors. Completion 
analytics that adjust to the mix of students 
served could also provide data that allow 
student advising to be better tailored to the 
individual student. Students are not all equally 
likely to succeed in college. Individual students 
vary in backgrounds, motivations, and abilities. 
Single performance measures for the college 
reflect the outcome for an average student 
at that college, and not a diverse range of 
individual students with their varying goals and 


conditions of learning. 


RESEARCH HYPOTHESIS 


Institutional research, college benchmarking, 

and student advising could all be improved if 
performance metrics were stratified along a 
continuous scale that reflects the range of diversity 


in a student's readiness to succeed. 


This paper provides supporting evidence for this 
hypothesis through consideration of five research 


questions (see below for definition of terms): 


1| Are 1+3 transfers successful at their transfer 


destination schools? 


2| How much does the student experience vary 
between college-ready and college-prep students? 


3| How does an ecosystem affect transfer behavior 
and student mix? 


4| Can we improve our understanding of college 
readiness beyond just a ready-or-not split? 


5| How do you show performance improvement 
when there are differences in student mix? 


TERMINOLOGY 


Ecosystem 


The local mix of higher education opportunities 
varies for students across colleges and universities 
in different regions. A different mix of competing 
colleges, educational programs, and enrollment 
capacities can result in one region having a much 
different mix of first-year students who begin at 
community college rather than those who begin at 
the local state university. Second- and third-year 
educational pathways and capacities can also vary 
by region, resulting in much different student 
transfer behavior. 


Three Types of Transfers: 2+2, 1+3, and 
Second-Chance Transfers 


For those seeking a 4-year degree, 2+2 transfers 
are students who begin college at a 2-year school 
and graduate there first before transferring to a 
4-year school. 


1+3 transfers are students who did well academically 
at their first 2-year school, but who did not 

graduate there. Instead they chose to transfer to 
another school to continue their education toward a 
4-year degree. 
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A third type of transfer are second-chance 
transfers, defined as students who did not do well 
academically at their first college and who earned 
very few credits before leaving. After leaving and 
perhaps even several years later, they found 

a second chance to continue their education 
elsewhere. Because second-chance transfers were 
not academically successful at their first school, 
these transfers should not be added into transfer- 
adjusted completion rates. 


College-Ready vs. College-Prep Students 


Not all high school graduates are academically 
prepared for college-level coursework. Community 
colleges sometimes use a placement test for their 
first-time-in-college students to identify those who 
would benefit from developmental coursework in 
basic reading, writing, and math before they take 
more-advanced college-level courses. 


THEORY 


Florida state colleges include some of the most 
respected community colleges in the country. Higher 
graduation rates lead to frequent performance 
awards, as well as the ongoing study by consultants 
and benchmarking organizations to learn the secrets 
to Florida's success. Valencia College in Florida is 
viewed as an exemplary role model, and that college 
won the inaugural Leah Meyer Austin Award in 

2009 and the inaugural Aspen Prize for Community 
College Excellence in 2011. 


Austin Community College in Texas, with high 
transfer rates and low graduation rates, provides 
an interesting contrast to Valencia College. 
Austin’s scorekeeping problem was addressed 
by the president of Valencia College in his article, 


“Rethinking the Completion Agenda” (Shugart, 2013). 
Shugart's article introduces the term “ecosystem” 

to define the higher education environment and 
how students behave in different ways across 
diverse ecosystems. He offers a compelling essay 
that explains how low graduation rates can be 
driven by high transfer rates attributable to the local 
environment (ecosystem) and are not due 

to a deficiency in college leadership. He suggests the 
need for more-collaborative research of 

student behavior across colleges, including the 
development of measures of the performance of the 


entire ecosystem. 


This paper builds on Dr. Shugart's working theories 
and provides additional supporting data. 


METHODOLOGY 


Case Studies 


Two community colleges are the case studies for this 
paper. These two schools operate in much different 

higher education environments, but both are greatly 
affected by the admissions practices of their local 


sister state universities. 


The Central Indiana region of lvy Tech Community 
College is in an environment where it is easy 

for students to transfer out before graduation. 
Neighboring state universities do not require a 
degree before they admit successful students as 
transfers. lvy Tech's retention and completion rates 


are scored very low. 


Valencia College in Central Florida is in an 
environment where state policy strongly discourages 
state universities from admitting community college 


transfers that did not first graduate. Valencia College 
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is an award-winning school with comparatively high 


rates of retention and graduation. 


Valencia College provides an ironic contrast in 
strategies to lvy Tech. The number one strategic 
priority at Valencia College is on student learning, 
and only secondarily on college completion 
performance metrics. Valencia leadership believes 
that greater learning will lead to better student 
performance. Conversely, with its low completion 
rates lvy Tech has always been compelled to focus 
its strategic priorities on improving retention and 
completion. How do you get students to stay 

until graduation, and how do you get students to 
graduate sooner so that their completion can be 
included within the limited number of years used for 


performance scorekeeping? 


Unique Sources of Data 


During our study both colleges had good sources 

of data and capable institutional research (IR) 
departments. This allowed us to analyze and 
compare the student experience in ways that go well 


beyond traditional college scorekeeping formulas. 


lvy Tech had built a data mart that tracked all first- 
time-in-college students over an 11-year period 
using data from the National Student Clearinghouse 
(2005-15). This allowed us to study transfers in 
terms of their long-term outcomes and to explore 
whether the individual student's choice to transfer 


was beneficial or harmful to that student. 


Valencia College routinely differentiated graduation 
rates between college-ready and college-prep 
students. lvy Tech adopted a similar approach for 
internal research and reporting. This differentiation 
enabled us to make comparisons in and across both 
schools on how student readiness to succeed affects 


graduation rate performance. 


Each community college had a close relationship 
with its local sister state university. This included 
coordinated research on issues of common 
interest. The admissions practices of local sister 
state universities affect the aptitude mix of students 
attending the local community college. In both 
Central Indiana and Central Florida, applicants 

who are not admitted to their state university as 
freshmen often start at the local community college 
with the goal of transferring to the local state 
university as soon as they are allowed to do so. Data 
on the aptitude or achievement levels of entering 
students at sister state universities are available 
online from websites that have been designed to aid 


students in college selection. 


Valencia’s sister university is the University of 
Central Florida (UCF), a school with high admission 
standards and low transfer-out rates. Due to rapid 
growth, UCF has always had very limited enrollment 
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ty to accept transfers-in as Sophomores. 


capaci 


lvy Tech's sister university is Indiana University- 
Purdue University Indianapolis (IUPUI), a regional 
campus shared by Indiana University and Purdue 
University. As a regional campus, IUPUI’s admissions 
standards are naturally lower than their two 
separate main campuses. IUPUI, with high rates 

of transfers out (including transfers to one of their 
main campuses), had the available enrollment 
capacity to easily accept students to transfer in well 


before those students had completed a degree. 


CAT-Scan Methodology 


The first question asked when considering a 
graduation rate is, “Why isn't it higher?” The simple 
answer is that the students did something else. 
Yet it is rare to see a graduation rate reported at 


the same time that all other student outcomes are 
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reported. It helps to graphically show all student 
outcomes in layers that add up to 100%. If you 
want more students to graduate, a 100% layered 
graph encourages you to consider how to get fewer 
students to do anything else. 


lvy Tech pioneered the use of CAT-scan (Cohorts 
Across Time) graphs to show the diverse ways 
students experience college. A CAT-scan graph 
simultaneously shows all student outcomes for all 
cohort outcome years. lvy Tech built these CAT- 
scans (Cornett & Hancock, 2015) by first creating 
a data mart that matched up all annual starting 
cohorts of students against each student's 
cohort-year outcomes as recorded in the National 


Student Clearinghouse. 


lvy Tech IR staff invested many months of effort to 
design and create their original CAT-scan data mart. 
Their effort increased every year as the variety of 
reported student outcomes expanded. Once the 
data mart had been created, the big reward was 
the ease by which precomputed outcomes could 
instantly be reported and disaggregated across any 
student group or research treatment variable. The 
data mart was delivered in a spreadsheet that 
could be explored using simple pivot-table methods. 
In a single afternoon, IR staff can produce scores of 
CAT-scan graphs. The hardest part is labeling the 
graphs and delivering the results so that users 

can understand all the information made available 
to them. 


Figure 1 is a basic 7-layer CAT-scan graph. Three 
different types of transfers are shown: 2+2, 1+3, and 
second-chance transfers. When studying a CAT- 
scan, it is useful to consider how results vary across 
cohort outcome years. In this example, by the end 
of Year 3 few students have graduated (3.8% = 1.2% 
+ 2.6%; areas in light green and dark green), while 
more have had a successful 1+3 transfer (11.3%; 


areas in yellow). 


Figure 1. This 7-layer CAT-scan combines 11 starting cohorts across 9 cohort outcome years. 


How Students Experience College 


7-Layer CAT-scan: Measures 7 major categories of student outcomes over time. 
Degree-Seeking only Ivy Tech - Central Region only 
100° College-Ready or not; Full or Part-Time; FTIC only; Combined 2004-2014 Fall starting cohorts; Outcomes to Fall 2015 
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Cohort Outcome Years (as of Fall) since first starting at lvy Tech 


Year 9 
n=6,081 


Year 4 
n=20,671 
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n=17,664 


Year 6 
n=15,196 


Year 7 
n=11,861 


Year 2 
n=28,177 
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n=31,584 


Year 3 
n=24,418 


IPEDS (Integrated Postsecondary Education 


Data System) serves as the national standard for 


Ina"1+3 Ecosystem" 


students are free to transfer at any time: 
2nd Chance transfers = those who fail with us, then transfer 

1+3 transfers = successful students who transfer with no degree 

2+2 transfers = successful students who graduate, then transfer 


Failed Academically 
Perhaps "life i 


Dropouts and Stopouts 


Successful students but "life intervened" 


Retained 
Continuing with us that outcome year 


"2+0" Graduates 
Did not transfer after graduating 


Academic "success" means the student 
earmed a final GPA of at least 2.00 or 
earned 12 total credit hours with us. 


academically show up as transfers in National 


Student Clearinghouse data. By Year 8 many initial 


benchmarking the performance of colleges. The college failures can be seen to have returned to 


National Center for Education Statistics (2020) school elsewhere to give college another try. 


recently began reporting community college 


completion outcomes out to 6 and 8 years instead Answering one research question often leads to 


of just 3 years, as was their previous standard. 


asking two new questions, and new questions 


quickly arise when first considering a 7-layer scan. 


As seen in Figure 1, lvy Tech's 8-year outcomes show 
an obvious improvement in graduation rates over 
3-year outcomes. Even so, after 8 years the 1+3 
transfers (17.9%) are still more common than all 
graduates (15.4% = 3.3% + 12.1%) from lvy Tech. 


Second-chance transfers also increase over time. 


In the first outcome year, few students who fail 
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The National Student Clearinghouse provides data 
that allow researchers to know much more about 


student outcomes. Additional research variables can 


be linked from a college’s own data warehouse. lvy 
Tech eventually built a 23-layer CAT-scan to study a 
much richer variety of student outcome questions 


(see Figure 2). 


Figure 2. A 23-layer CAT-scan shows many more types of outcomes than a basic 7-layer CAT-scan. 


. Within this 23-Layer CAT-scan, 
23-Layer CAT-scan (Cohorts Across Time) aici are three types of transfers: 
23-Layer CAT-scan: Measures a more complete combination of student experiences over time. — 2nd Chance transfers = those who fail with us, then transfer 


Degree-Seeking only Ivy Tech - Central Region only 1+3 transfers = successful students who transfer with no degree 
College-Ready or not; Full or Part-Time; FTIC only; Combined 2004-2014 Fall starting cohorts; Outcomes to Fall 2015 2+2 transfers = successful students who graduate, then transfer 
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n=31,584 n=28,177 n=24,418 n=20,671 n=17,664 n=15,196 n=11,861 n=8,650 n=6,081 Academic "success" means the student 
Cohort Outcome Years (as of Fall) since first starting at lvy Tech earmed a final GPA of at least 2.00 or 
earned 12 total credit hours with us. 
In this 23-layer CAT-scan, 1+3 transfers are For example, Indiana considered the creation of a 
disaggregated into three transfer-destination 30-credit transfer certificate that could be earned by 
outcomes: students who have graduated, those students who completed the right mix of freshman- 
continuing in school somewhere, and those who have level courses. That certificate would be interpreted 


dropped out. Each of these three categories is further by scorekeepers as a type of pretransfer completion 


split between those who earned at least 30 credits credential. The use of 30-credits-earned layers in 
before leaving and those who transferred out before the 23-layer CAT-scan shows how the introduction 
earning 30 credits. This combination of six outcomes of a 30-credit transfer certificate might affect college 
within the 1+3 transfer type allows us to explore a completion rates, and how these rates would vary 
wide range of what-if scorekeeping questions. depending on how many cohort years are counted 


in the scorekeeping. 
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R ES EARC H Q U ESTI O N 4: who hoe a ie a “ 
ARE 14+3 TRANSFERS completion and persistence after leaving lvy Tech: 
SU CC ESS FU L AT TH E | R lvy Tech studied graduation, retention, and dropout 
TRANS F E R D ESTI NATI O N rates for 1+3 transfers as of the eighth cohort 
SC H O O |S? year after starting at lvy Tech. These results were 


disaggregated across categories for lvy Tech grade 
Although transfers before graduation damage a 

point average (GPA) and credit hours earned 

(Cornett et al., 2016) (see Figure 3). lvy Tech GPA 


appears to have a strong influence on success 


college’s reported performance, the important 
question is whether 1+3 transfers hurt a student's 


performance. Do academically successful students 
after transfer. 


Figure 3. 1+3 transfer outcomes at destination schools show high rates of graduation and 
persistence. 


1+3 Tra n sfe r Outco me M ix vs. lvy Tech Hours and GPA Groupings 


8th year transfer outcomes at other schools after transfering from Ivy Tech Central Region 
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Overall, students who transferred out 1+3 style 
earned a degree somewhere else at a rate of 49.2%. 
This is more than three times the 15.5% rate that 


all students starting at lvy Tech graduate from Ivy 


Tech with an associate degree. Not all 1+3 transfers 
complete, but most (76.3%) either graduate (49.2%) 


or are still in school (27.1%) when measured over an 


8-year horizon since first starting at lvy Tech. 


The effect of additional hours earned at lvy Tech 

on success after transfer is uncertain. This study 
included both full-time and part-time students. 
Part-time students naturally need more years to 
graduate than full-time students, and many part- 
time students persist in school as long as needed to 


complete their degree. 


With relatively high rates of graduation and 
persistence in school after transfer, it is difficult 
to regard 1+3 transfers as mistakes from a 


student's perspective. 


RESEARCH QUESTION 2: 
HOW MUCH DOES THE 
STUDENT EXPERIENCE 
VARY BETWEEN COLLEGE- 
READY AND COLLEGE- 
PREP STUDENTS? 


Community colleges have open admissions 
standards. All high school graduates who apply are 
admitted. This includes the full range of students 

in terms of their readiness to succeed in college, 
ranging from straight-A students to those with poor 
grades who barely graduated. 
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It seems intuitively obvious that students who are 

better prepared for college will generally succeed in 
college at higher rates; this hypothesis still needs to 
be researched and measured. How much better do 


college-ready students perform? 


In years past, Florida and Indiana community 
colleges required placement tests of all their 
entering students. College-prep students, often 
referred to as developmental students, are students 
whose placement test scores show them to not 

be academically ready for college. They graduated 
from high school and seek a college degree. They 
are advised to begin community college by taking 
remedial courses designed to better prepare 

them in basic reading, writing, and math before 


transitioning to college-level coursework. 


State laws in Florida and Indiana no longer allow 
such tests to be required of all community college 
students. It is now more difficult to research 
student success, but for the time frame of this 
study it was possible to split students into the two 
separate research groups of college-ready and 


college-prep students. 


Using their CAT-scan data mart, lvy Tech outcomes 
were easily split between college-prep and college- 
ready students (see Figure 4). Internal graduation 
rates were computed over an 8-year horizon for a 
combination of part-time and full-time first-time-in- 


college degree-seeking students. 


Figure 4. College-prep and college-ready students have different experiences. 


Ina "1+3 Ecosystem" 
students are free to transfer at any time: 


The College-Ready Student Experience 


7-Layer CAT-scan: Measures 7 major categories of student outcomes over time. 2nd Chance transfers = those who fail with us, then transfer 
Degree-Seeking only Ivy Tech - Central Region only 1+3 transfers = successful students who transfer with no degree 
100% College-Ready only; Full or Part-Time; FTIC only; Combined 2004-2014 Fall starting cohorts; Outcomes to Fall 2015 2+2 transfers = successful students who graduate, then transfer 
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Ina "1+3 Ecosystem" 
students are free to transfer at any time: 


The College-Prep Student Experience 


7-Layer CAT-scan: Measures 7 major categories of student outcomes over time. 2nd Chance transfers = those who fail with us, then transfer 
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College-ready students at lvy Tech had more than 
twice the graduation rate (25.3% = 21.6% + 3.7%) as 
college-prep students (11.0% = 7.8% + 3.2%). 


Valencia College provides an interactive tool on their 
public website (Valencia College, 2018) that allows 
8-year graduation rates to be reported the same way 
that lvy Tech reported theirs. College-ready students 
at Valencia had almost twice the graduation rate 
(48.9%) as college-prep students (27.0%). 


In both colleges the data show that being college- 
ready is strongly associated with higher graduation 


rates, with roughly twice the rate. 


The experiences of nongraduating students are 
also worth analyzing and improving. College-ready 
students experience college in much different 
patterns over time than college-prep students. 

vy Tech CAT-scans invite consideration of many 
alternative outcomes including retention, 1+3 


transfers, drop-outs, second-chance transfers, and 


failed students (see Figure 4). 


Researchers sometimes recommend a combined 
success metric to include graduates plus transfers. 
The Institute for Higher Education Policy (Janice 

& Voight, 2016) recommended limiting transfers 

to programs that are longer than the previous 
program, and did not distinguish between 1+3 
transfers and second-chance transfers. Regardless 
of the transfer definition used, adding transfers to 
graduation rates may prove useful in narrowing 


performance gaps across ecosystems. 


Figure 4 illustrates what happens when adding 1+3 
transfers (yellow areas) to graduates (green areas). 
After 8 years lvy Tech college-ready students show 
a 1+3 transfer-adjusted completion rate of 46% 


compared to college-prep students at 28%. 
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Scorekeeping analytics are often the criteria used to 
evaluate the results from research studies designed 
to improve student success. When all transfers 

are scored as failures, and when student aptitude 
mix is not considered, research studies can lead to 
inconclusive results or even incorrect results. 
Better scorekeeping could lead to more-accurate 


research studies. 


Regardless of the scorekeeping formula used, the 
student experience and completion success rates 
vary widely as a function of college readiness. These 
are not identical populations where the mix of 
students of each type has no effect on completion 


rate scorekeeping. 


RESEARCH QUESTION 3: 
HOW DOES AN 
ECOSYSTEM AFFECT 
TRANSFER BEHAVIOR AND 
STUDENT MIX? 


University admissions practices determine the local 


community college ecosystem. 


Given high rates of transfer, and corresponding low 
rates of graduation, it is an all-too-common mistake 
to think that 1+3 colleges are to blame for choosing 
a 1+3 transfer strategy. External consultants who 
have closely studied Florida colleges as their role 
models advise against supporting the transfer goals 
of students who do not wish to stay until graduation. 
Without regard to the effects of a transfer-oriented 


re 


ecosystem, consultants challenge college leadership 


to communicate to their students in ways to try to 
create a stronger value proposition for students to 


complete an associate degree before transfer. 
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The theory that transfers a 


re something out of place 


and to be avoided is countered in recent research 


(Shapiro et al., 2016). That paper concludes, 


“Multiple-institution attendance is common. Simply 


recommending to students that they not change 


institutions and promoting 


the benefits of single 


institution attendance does not seem to be useful 


any longer” (Shapiro et al., 2016, 25). 


The 2-year colleges do not 


control student transfer 


opportunities. Rather, it is the admission practices 


of the 4-year universities that determine whether 


community colleges operate within a 1+3 or 2+2 


ecosystem. When given ac 


hoice, many students will 


transfer early in order to spend more time at the 


school at which they plan to earn their bachelor's 


degree. Community colleges do not get to choose 


their ecosystem as a Strate 


gy. The only choice 


college leadership gets to make is how much to 


advise students on their transfer choices and what 


courses to advise students 


to take before they leave. 


Ecosystems behave much differently when the 


sophomore year choice is open for individual 


students to decide for themselves. Most programs 


at IUPUI require only 28 co 


llege credits earned 


before students can be admitted as transfers, and 


some programs require on 
all transfers-out are before 
70% of all transfers-out to 


graduation. In Valencia Col 


ly 24. In lvy Tech 90% of 
graduation, and over 
UPUI are before 


ege, the pattern is 


reversed, and 86% of all transfers-out from Valencia 


to UCF are after graduation 


Knowledge Management [I 
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(UCF Institutional 
KM], 2011-15). 


The pairing between the local community college 
with its nearby sister public university greatly 
influences the student mix in terms of student 
aptitude and college readiness, especially in isolated 
communities. When the local sister university 
admits only the best-qualified students, the aptitude 
mix of local community college students must 


correspondingly increase to fill the educational gap. 


Many students are place bound and have a limited 
choice as to where they can go to college. Education 
deserts are places where there is no local option 
available for a place-bound student to attend a 
college of the right fit (Hillman & Weichman, 2016). 
Capacity constraints and the admissions practices 
for a sister university can effectively create a local 
education desert for students who do not meet 
academic standards. In 2+2 environments, capacity 
constraints can cause many excellent college-ready 
but place-bound students to begin at a community 
college. These students could have started at 

the local university if they resided in a different 


ecosystem. 


Figure 5 shows how the range of freshman SAT 
scores vary across three universities. The red curve 
shows the cumulative distribution of nationwide 
SAT percentiles (College Board, 2017). Ranges of 
freshman admission 25th to 75th percentiles are 
shown as boxes on this curve (Prep Scholar, 2018). 
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Figure 5. Admissions standards vary widely for the local sister universities of community colleges. 


University SAT Ranges (25th to 75th Percentiles) 
versus Overall SAT Percentiles 
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Comparing the 25th to 75th percentile SAT scores 
for freshman admissions, UCF (70%-92%) has a 
much better-prepared mix of entering students 
than IUPUI (37%-76%). The University of Florida 
UF) (82%-96%) has an even better-prepared mix 
of students than UCF. UCF is the sister university 


to 2011 Aspen Prize-winning Valencia College in 


Orlando, Florida, while UF is the sister university 


4. 


to the 2015 Aspen champion, Santa Fe College in 


Gainesville, Florida. 


The 25th to 75th percentile range of SAT scores 
barely overlap between IUPUI and UCF. Some of 
the top half of students at IUPUI would only rank 


within the lower quartile of UCF’s freshman class. 
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Nevertheless, these schools appropriately serve 
their role within their educational ecosystem. Given 
UCF's size and capacity constraints, they cannot 
grow larger, so they must turn away more and more 
freshman applicants. Fortunately, these students can 
attend Valencia College (and other local community 
colleges) where they are guaranteed a direct- 


connect admission to UCF on graduation. 


Community colleges also serve many elite aptitude 
students, and offer honors programs to target 

these students. Universities compete aggressively 
for the best-prepared students and are less likely 


to compete for those students who fall below their 


4 


25th percentile—especially when the university is 


capacity constrained. Students who are not admitted 
to their local public university can instead attend 
their local community college. 


Based on the 25th percentile of sister university SAT 
scores, the aptitude mix of college-ready students 
available for Valencia College to serve (below 70th 
percentile) is a much wider range of middle aptitude 


students than those left available to Ivy Tech (below 
37th percentile) (see Figure 6). Consequently, the 
Valencia college-ready student mix is wider and 
shifted to the right on the horizontal SAT scale 
compared to lv Tech. For college-prep students, a 
right-shift effect is less certain, but could still occur 
because IUPUI's lower quartile of students can more 
easily dip into the college-prep market. 


Figure 6. The college-ready students who enroll at Valencia should be shifted to the right of Ivy 


Tech's college-ready mix. 


College-Ready Students available to be served 
by Ivy Tech and by Valencia College 
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These data show how the mix of students starting 
college at Valencia should be much better prepared 
to succeed than those starting college at lvy Tech. 
This does not mean that either college should try to 
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change the mix of students they serve, but it does 
mean that scorekeepers need to work harder at 
completion scorekeeping to control for mix effects. 


RESEARCH QUESTION 4: 
CAN WE IMPROVE OUR 
UNDERSTANDING OF 
COLLEGE READINESS 
BEYOND JUST A READY- 
OR-NOT SPLIT? 


The use of placement tests to flag community 

college students as being ready (or not ready) for 
college-level coursework is a small step toward 
understanding and advising students of diverse 
backgrounds. Young adults enrolling in college resent 


being categorized as not yet ready, and there are 


better ways to evaluate student readiness to succeed. 


Colleges and universities with admissions standards 
often require standardized aptitude or achievement 
tests to guide their choice of student applicants to 
accept. High school GPAs and percentiles are also 


widely used in the admissions process. 


It seems logical that students who do well in high 


school will be better prepared to succeed in college. 
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An in-depth study (Balfanz et al., 2016) confirms that 
students who graduate from high school with higher 
GPAs are much more likely to succeed in college. 
The Balfanz study also found that high school GPA is 
a better predictor of college success than aptitude 
tests alone. Using GPA as an indicator of college 
readiness is useful because aptitude and placement 
tests are not available for all college students, but all 


students have a high school GPA. 


In theory, college completion success rates could 
be modeled to show a steady increase as a function 
of whatever student aptitude measurement scale is 
used (test scores, GPA, or any logical combination). 
The green curve in Figure 7 is an illustration of what 
such a curve might look like. The horizontal axis is 
defined to be high school GPA percentiles (although 
it could be any other aptitude scale). The vertical 
scale is modeled as the percent of students likely 
to succeed in college. For Ivy Tech and Valencia 
colleges, the 1+3 transfer-adjusted graduation 

rate is plotted against this scale, and split between 
college-prep (CP) and college-ready (CR) students. 


Percent of All Students 


Figure 7. In theory, 1+3 transfer-adjusted graduation rates could be plotted against a national 


benchmark curve. 
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It is not known whether curves like this are available 
from any published source. Therefore, the exact 
shape of this green curve is only an educated guess. 
To aid in modeling this curve, data on student 
aptitude and success rates are parameterized 
against four market segments. 


Elite universities have graduation rates that come 
close to the high graduation rates shown for the 
elite-aptitude student segment. Middle-aptitude and 
college-prep students split the middle part of the 
graph. Students labeled as “not interested” in college 
are the least likely to succeed. 


The Balfanz study (Balfanz et al., 2016) provides data 
that help to define the widths of the four student 
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market segments illustrated in Figure 7. Twenty-two 
percent of students did not attend college within 10 
years of high school graduation (the not-interested 
market segment). Another 22% of high school 
graduates do enroll in college but are not strongly 
prepared and are at risk of failing (the college-prep 
market segment). The rest are college-ready and are 
further split between the top 20% (the elite-aptitude 
market segment) and everyone else in the middle 
(the middle-aptitude market segment). 


More research is needed, but this proposed curve 
is useful to illustrate important concepts. The shape 
of this green curve was intentionally designed to fit 
the data available when comparing 1+3 transfer- 
adjusted graduation rates across Valencia and 
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lvy Tech colleges. Neither college had high school 
GPA data on a student-by-student basis, but their 
placement test split for college readiness allows 
two groups of students to be considered for 


each college. 


On the horizontal axis, the college-prep and college- 
ready splits for each school are placed separately 
within their horizontal market segments. The exact 
horizontal percentile placement for these schools is 
unknown, but the relative market positioning to 
their sister universities suggests that Valencia is 
shifted to the right of Ivy Tech, especially for college- 
ready students. 


On the vertical axis, each college's 8-year actual 
graduation rates are shown as red triangles. The 
green circles are meant to show the student's 
completion perspective in terms of a 1+3 transfer- 
adjusted graduation rate. The 1+3 transfer rates 
are not available for Valencia College, but their 
ecosystem discourages transfers. For the sake of 
illustration, a rough estimate would be two-thirds of 


the lvy Tech rates. 


There are a lot of unknowns in Figure 7, but it does 
illustrate what could be studied if enough data were 


gathered along a national benchmark curve for 1+3 
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transfer—-adjusted graduation rates. Some important 
things to look for when comparing colleges across 


diverse ecosystems are the following: 


1| Acomparison of college-prep against college- 
ready students should reveal major differences 
in graduation rates, transfer rates, and transfer- 
adjusted rates. These performance differences 
should be heavily influenced by differing levels 


of college readiness. 


2| Low graduation rates for a college (such as lvy 
Tech) might be explainable based on the mix of 
students served and the transfer-rates expected 


within its ecosystem. 


3| High graduation rate performance for a college 
(such as Valencia) would be further validated 
by showing transfer-adjusted graduation rates 


raised above a national benchmark curve. 


Figure 8 adds additional data to illustrate the relative 
positioning of three sister universities along a 
student aptitude benchmark curve. The exact shape 
of this curve and exact placement of colleges and 
universities on this graph is not currently known but 
could be calculated at the state level by authorities 
with access to both high school GPAs and college 
student outcome records. 
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Percent of All Students 


Figure 8. A national benchmark curve would allow the performance of community colleges to be 


compared to their sister universities. 
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UCF and UF are placed to the right on this graph 
based on their freshman SAT quartiles (Prep Scholar, 
2018). UCF and UF have relatively high graduation 
rates (College Navigator, 2018). There is no public 
source for 1+3 style transfer rates. As premier 
schools, UCF and UF transfer-out rates may be 
mostly second-chance transfers. Assuming their 

1+3 style transfer rate is low, we use their 
graduation rates alone to compare to the green 


benchmark curve. 


It is harder to place IUPUI on this curve. They have 
a high rate (31%) of reported total transfers out 


(College Navigator, 2018). These transfers will include 
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both second-chance transfers and 1+3 style transfers 
including students using this regional campus as 

a transfer stepping-stone to Indiana University or 
Purdue University main campuses. IUPUI's freshman 
SAT quartiles place it somewhere just above the 
middle on the student aptitude horizontal scale. 

It seems likely that IUPUI's 1+3 transfer-adjusted 
graduation rate would fall very close to where it 


would be expected on the green curve. 


Differences in completion performance across all 
colleges and universities could be better interpreted 
using a national benchmark curve based ona 


student aptitude scale. 
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RESEARCH QUESTION 5: 
HOW DO YOU SHOW 
PERFORMANCE 
IMPROVEMENT WHEN 
THERE ARE DIFFERENCES 
IN STUDENT MIX? 


Performance means a vertical lift in student success, 


not a right shift in student mix. 


When modeling performance against a student 
aptitude scale, performance gaps across groups 


of students can be narrowed and more carefully 


interpreted. Any vertical movement upward 
is progress and any performance above the 


benchmark curve is worthy of recognition. 


Figure 9 illustrates how vertical lift could someday be 
reported as a continuum against a benchmark curve, 
and not just as two data points for the average 
success rate split between college-prep and college- 
ready students. An ideal approach might be to 
report success at 5-percentile increments along the 
horizontal axis. (The blue line is not real data for any 
college but demonstrates what a line might look like 


for a college like Valencia.) 


Figure 9. Vertical lift is the value added to success that the college offers a student. 


Performance should really mean the vertical "lift" 
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It is possible, as shown here, that a college's 
performance lift with students could vary along the 
horizontal axis. In this theoretical scenario, the college 
provides greater vertical lift in success for those 
students who come to college less well prepared. 


For students trying to decide where to attend 
college, an advisor could use graphs like these to 
coach a student based on where that individual 
student sits on the horizontal axis. This is much 
better information than just the average graduation 
rate for a college or university. 


RECOMMENDATIONS FOR 
FUTURE RESEARCH 


More research is needed to validate the findings 

of this study. Many more community colleges and 
their sister universities need to be analyzed across a 
broad range of ecosystems. 


This paper hypothesizes that high school GPAs and 
their percentiles might guide the interpretation of 
student performance, but necessary testing has 

not yet been done. Neither Valencia College nor 

lvy Tech had high school GPA data available to use 
on a student-by-student basis. Privacy laws make it 
difficult to access these data, so testing their use may 
need to be conducted on a state or national level. 


When studies like this one are done in the future, 
researchers might consider exploring the following 
research questions: 


WHAT IS THE BEST WAY TO MEASURE 
VERTICAL LIFT WHILE CONTROLLING FOR 
HORIZONTAL SHIFT? 


A college's performance should be judged by the 
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vertical lift its students exhibit compared to a 
benchmark curve, not the right shift that occurs 
when one college is able to recruit a better mix 

of students. This is true not only of community 
colleges but also of 4-year schools. Middle-aptitude- 
serving universities cannot be expected to have 
graduation rates as high as the elite-aptitude-serving 
universities. Any college or university could have its 
students stratified along a high school GPA scale so 
as not to judge a whole college on a single number 
that presumes all students are equally well prepared 
to succeed. 


HOW USEFUL ARE 1+3 TRANSFER-ADJUSTED 
GRADUATION RATES? 


Using a 1+3 transfer—adjusted graduation rate 
represents the student perspective on transfers as a 
positive outcome, but this metric may prove to have 
its own limitations. Perhaps a better scorekeeping 
formula would add in student retention and not 

just transfers. Can ecosystems be classified, and 
separate benchmarks developed for each type of 
environment? More research is needed on how 
different scorekeeping formulas perform when 
measured across diverse ecosystems. The important 
thing is to test those formulas against a horizontal 
scale that allows students to be differentiated based 
on their readiness to succeed. 


CAN WE BUILD BETTER DECISION-SUPPORT 
TOOLS FOR USE BY STUDENTS AND THEIR 
ADVISORS? 


When students are advised in their college choices, 
they should be given information tailored to their 
level of college readiness, and not just a single 
performance rate for the average student at a 
college. Students have a range of outcomes that 
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they might experience over time, as shown by CAT- 
scan graphs. Perhaps someday an online tool could 
be provided to students that asks them for their 
high school GPA and a particular college they would 
like to consider. The system could then respond with 
a CAT-scan-style 7-layer range of outcomes (adding 
up to 100%) that students with GPAs like them 
experience at that college by the end of 8 years. 
The student does not need a performance score for 
the college, but would be better advised to consider 
all the possible outcomes that can and do happen 


for similar students. 


CAN WE IMPROVE THE CONVERSATION 
ABOUT BEST PRACTICES AND IMPROVEMENT 
OPPORTUNITIES? 


Researchers, benchmarking organizations, and 
consultants to colleges are continually seeking to 
discover those best practice initiatives in use at high- 
performing colleges that all colleges should consider 
adopting. Unfortunately, they are misinformed by 
triple-digit gaps in completion analytics as reported 
by scorekeepers in the past. Combining many 

small benefits from best practices in classroom 
learning success, it seems plausible to achieve a 

5% to 10% improvement in overall completion 

rate. Best-practice student initiatives cannot cause 

a ten-fold improvement as traditional IPEDS-style 
3-year scorekeeping implies. Figure 10 shows how 
benchmarking and scorekeeping is typically done 
(National Center for Education Statistics, 2015). 
IPEDS-style graduation rates like these differ to such 


an extreme that the results lack face validity. 


Figure 10. Single performance scores for a college leave so much unexplained. 
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AUTHOR'S BACKGROUND 
AND RESEARCH BIAS 


Before serving as IR director at lvy Tech, | had 

the privilege to serve in the same role at Valencia 
College. This meant that | first worked at one of 
the nation’s most prestigious community colleges 
with high rates of academic success. | suddenly 
found myself working at a college where we had 
inexplicably low completion rates according to 
PEDS rules. 


arrived at lvy Tech believing that transfers before 


graduation should never be thought of as a success 


for a college. By the time | left, | had completely 
reversed my thinking. Students in both Indiana and 


Florida were making the right choices for themselves 


given the options available to them. If a successful 
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student earns the opportunity to transfer to a 
better-funded college, especially if their goal is to 
earn their 4-year degree at that specific school, that 
is the choice | would make (or would want my child 


to make). 


Both Ivy Tech and Valencia naturally serve the role 
they need to play within their ecosystems. Neither 
the colleges nor their ecosystems need fixing. The 
mix of students these colleges serve also does not 
need fixing. It is the scorekeeping that is broken, 


and everyone needs to work much harder at doing 


completion analytics better. It should be possible to 
improve our research, benchmarking, and advising 
through a deeper understanding of how diverse 
students experience college differently across 


diverse ecosystems. 
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